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Image smear, produced by the shutter-less operation of frame transfer CCD detectors, can be detrimental 
for many imaging applications. Existing algorithms used to numerically remove smear, do not contem¬ 
plate cases where intensity levels change considerably between consecutive frame exposures. In this 
report we reformulate the smearing model to include specific variations of the sensor illumination. The 
corresponding desmearing expression and its noise properties are also presented and demonstrated in the 
context of fast imaging poiarimetry. © 2015 Optical Society of America 
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1. INTRODUCTION 

CCD sensors with frame transfer architecture are currently of 
wide spread use in many imaging applications. A frame trans¬ 
fer CCD collects photo-charges in a light sensitive area during 
exposure time to then rapidly clock them towards a shielded, 
frame storage area where readout is performed [1]. This kind 
of detector readily become an interesting candidate when frame 
rate, duty cycle and optical fill factor need to be simultaneously 
maximized. 

If the CCD is continuously illuminated during frame trans¬ 
fer, the resulting image will be smeared reducing its contrast 
and spatial resolution [1]. This artifact can be a draw back in 
many applications [2-5] becoming particularly relevant when 
high frame rate and duty cycle are required. Given its gen¬ 
eral applicability, low cost and null effect on duty cycle among 
others, numeric algorithms for post-facto correction of image 
smear are an attractive alternative to the classic mechanical or 
optoelectronic shutter systems. 

The first desmearing algorithms were introduced by Powell 
et al. [6], for standard, charge flush and reverse clocking CCD 
operation modes, and Ruyten [7] only for charge flush mode. In 
both of the above-mentioned works authors assumed, among 
others, that pixel saturation has not taken place, noise in the im¬ 
ages is negligible and the image illuminating the CCD does not 
change within exposure and frame transfer time. The first of 
these assumptions was investigated by Knox [8], who showed 


that smearing can be used in specific cases to recover the val¬ 
ues of saturated pixels in the image. More recently, different 
smearing correction techniques have been developed that are 
optimal for specific imaging applications [9,10] or used for real 
time correction ]11]. 

In all the above-mentioned works, the original assumption 
in ]6,8] of constant illumination is maintained. In this report we 
firstly develop the smearing model to treat cases where sensor 
illumination cannot be considered constant between consecu¬ 
tive frames, provided the photon flux is constant between tran¬ 
sitions, and the transitions occur in synchronization with the 
frame transfers. Further we assume the flux transition profile 
is symmetric with respect to its inflection point (Section 2). Sec¬ 
ondly we study the corresponding desmearing algorithm and 
its conditioning for the cases of non-periodic and periodic vari¬ 
able scenes (Section 3). 

The results of this work may be of utility to any application 
where the scene is highly changing in synchronization with the 
detector readout. An example are the fast polarimetric measure¬ 
ments we use to demonstrate our results (Section 4). 

2. FRAME transfer MODEL WITH VARIABLE ILLUMI¬ 
NATION 

We will assume frame transfer is performed in columns direc¬ 
tion and neither pixel saturation nor blooming has occurred. 
Under these assumptions the analyses in the present and fol- 
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lowing sections are restricted to a single sensor column without 
losing generality. 

Fig. 1 illustrates the operation in standard mode of a CCD 
with M rows during the acquisition of a single pixel in row no. 
m of frame no. k. The upper sketch represents the position of 
charge well no. m within the column. The lower profile ex¬ 
emplifies the values of the photo charges flux in pixel no. m as 
function of time, Sm(t)- The considered frame extends from 
time instants tf. to tj^_^_i with — tj. being the reciprocal of 
the CCD frame rate. The sensor is assumed to be permanently 
illuminated. The black boxes delimit the intervals where frame 
transfers are performed. 


Position 



the value of the smeared signal acquired in pixel no. m of frame 
no. k can be expressed as, 

M-l 1 

= X] + 8mteSm +-gmtsiSm ++ 

j=m+l 

m-l M-l 

dm{te + ts)+ km + flit Xj ‘^j' 

;=0 ;=0 

j^m 

where the time constants gj, dj and bj denote the gain, dark cur¬ 
rent flux and bias of pixel no. j respectively. The two ad-hoc 
coefficients and r 2 , have been introduced to facilitate experi¬ 
mental tuning of the model, e.g. to: contemplate possible differ¬ 
ences in the photo-charges-generation efficiencies of the static 
and transferring clocking states; compensate the effects of syn¬ 
chronization errors between the CCD readout and flux switch¬ 
ing; model other CCD operation modes; etc. 

The simple form of the third term in the right hand side of 
Eq. 1, which accounts for the signal accumulated during ts, is a 
direct consequence of the symmetry restriction imposed on the 
flux transition profile. The later has been defined by the applica¬ 
tion that motivated this work. In spite of that, notice it also cov¬ 
ers other cases of possible practical interest like very fast switch¬ 
ing (step-like profile), linear switching (ramp-like profile), con¬ 
stant illumination, etc. 

In the following we will assume the acquired images have 
been offset corrected using the dark frame, 

M-l 

Dm — ^m{le “F Is) “F “F tf ^ dj, (2) 

j=0 

j^m 


Fig.1. Process required to acquire pixel no. m of frame no. k, 
in a frame transfer CCD with M rows operating in standard 
mode. Upper sketch illustrates the position within the column 
of charge well no. m during frame acquisition. The lower pro¬ 
file exemplifies the model we adopted for the photo charges 
flux at pixel no. m, Sm{t)- Namely constant during frame trans¬ 
fer (black boxes with duration Mlf) and exposure time (te), 
and variable during switching time (ts). The transition profile 
is assumed symmetric with respect to the point (black circle) 
with coordinates {t^+i — ts/2, (S^ -F S^+^)/2}. See the text for 
additional details. 


Acquisition of frame no. k starts during the transfer of frame 
no. k — 1. An empty charge well is shifted from the top of the ar¬ 
ray to position no. m during a fraction of the total frame transfer 
time given by (M — m -t- l)tf, where ft is the period of the charge 
transfer clock. After this, static accumulation at position no. m 
takes place during the exposure time, tg. The flux is assumed to 
be constant and equal to during frame transfer and expo¬ 
sure time. A transition of the flux to its constant value in the 
next frame, is then contemplated. The flux transition oc¬ 
curs during the switching time tg, with the charge well still at 
position no. m. The flux transition profile is assumed to be 
symmetric with respect to the point with Cartesian coordinates 
{tk+i — fs/2, (S^ -F S^^)/2} in Fig. 1 (black circle). In the final 
step, once the flux is stable again, the charge well is transfered 
to the storage area of the sensor. 

For the above-described process and neglecting noise contri¬ 
butions (see Sections 2A and 3A), pixel saturation, charge trans¬ 
fer inefficiencies and instabilities of the charge transfer clock. 


which in practice can be obtained by averaging a long sequence 
of dark images, i.e. with Sm{t) = 0. 

In addition, let us define the unsmeared signal acquired dur¬ 
ing the static and constant illumination exposure as. 


— 8mteS^ 


(3) 


and constants. 


a = 





and 



(4) 


Note that Eq. 3 implies any gain table corrections should be 
applied after the desmearing process (see Section 3). 

Using Eq. 1-4 and redefining Yj^ — Dm as Ym, we can write a 
smeared column of the sensor in matrix form as, 

yk ^ ay*" -F BY*'+\ (5) 


where, Y*^ = 
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and the superscript T denotes the transpose. 

Note that, an analogue analysis can be carried out for the 
case where frame transfer takes place before the flux switching. 
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deriving in a causal difference equation also with matrix coeffi¬ 
cients A and B. 

In case the value of ft it is not known and can not be directly 
measured, some techniques have been developed to allow its 
experimental estimation for specific scenarios [7,12]. 

When operating the CCD in charge flush mode, the charge 
wells of the light sensitive area are flushed after the frame trans¬ 
fer is finished. Eq. 5 can be modified to model this scenario by 
doing r-y = 0. In reverse clocking mode the wells of the light sen¬ 
sitive area are transfered to a charge drain, located at the top of 
the sensor, by means of a reverse clocking. The latter is done 
after the usual frame transfer. To consider this case in Eq. 5, A 
has to be replaced by A^ and ry and r 2 adjusted appropriately 
if the clock rates, or charge transfer inefficiencies, of the usual 
frame transfer and reverse sweep are different. 

A. Noise properties of the smeared images 

To investigate the noise properties of the smearing process, we 
consider the unsmeared columns in Eq. 5 as independent ran¬ 
dom vectors. We also include an independent, additive noise 
term, to the right hand side of Eq. 5 to account for extra 
noise sources added during or after the charge wells are read, 
e.g. readout noise. Eor the above-described scenario one can 
show that, 

VarjY*"] = AVarjY*"]A^ -b BVar[Y''+^]B^ -b VarlN*"] 
Cov[Y^Y*'+l] = BVarjY^'+ijA^ 

where Var and Cov represent the variance-covariance and cross¬ 
covariance operations respectively. Note that Eq. 6 evidences 
the spatial and temporal correlation introduced by the smear¬ 
ing process. 

We further assume in Eq. 6 , that Var[Y*^+^] = 7 Var[Y*^] with 
7 € R, e.g. due to photon noise variation between frames with 
very different illumination levels. Then, a simple upper limit 
for the noise degradation can be written as follows, 

I Var[Y'‘] \<r]\ VarjY'"] | -b | VarjN'"] |, (7) 

where | . | denotes the euclidean matrix norm and, 

t] = (1 + a + 2Mdy/n)^ -b 7 (a + lM52lnY for M S> 1. ( 8 ) 

In the latter we used the fact that A= (1-ba — (ii)I-b<5iA and 
B = (a — ^ 2)1 + where I is the MxM identity matrix and 
A an MxM upper triangular matrix with A,-y = 1 for j > i and 
A,yy = 0 for i < i. 

Eig. 2 shows curves of y/tj versus MS for different values 
of a and 7 (with ry = r 2 = 1). When interpreting Eigure 2, 
and similar ones in the next section, recall that by definition 
for a fixed frame rate and exposure time, the sum 2oc + MS is 
constant. 

3. IMAGE RESTORATION 
A. Non-periodic iiiumination 

In this section we discuss the solution of Eq. 5 to 
an arbitrary time series of smeared columns, Y = 

I^(yO)^, (y 1)^, ..., (Y-*^^i)’r’j , with ft > 2 e N. Given 
the shape of matrices A and B, the restored columns, 

Y = |^(Y°)^, (Y^)^,..., , can be efficiently ob¬ 

tained by solving iteratively from pixel no. M — 1 to 0 and 
frame no. ft — 1 to 0 if the final condition, Y^, is known. 



Fig. 2. Upper boundary for the noise degradation due to the 
smearing process, versus the parameters product MS. The 
curves in each set, continuous for a = 0 . 0 % and dotted for 
a = 10 . 0 %, correspond to 7 = 0 . 1 , 1.0 and 10.0 from bottom to 
top respectively. See Eq. 7-8 for extra details. 



Fig. 3. Upper boundary for the propagation of errors in the fi¬ 
nal condition of Eq. 5, versus index of the restored frame. The 
curves in each set, continuous for a = 0 . 0 % and dotted for 
a = 10.0%, correspond to MS = 0.5,7.0,13.5 and 20.0% from 
left to right respectively 

. Note the vertical axis is in logarithmic scale. See Eq. 9 for extra 
details. 

The difficulties to obtain Y^ can be avoided after considering 
the general solution of Eq. 5, 

K-k 

Y*" = X] (9) 

M 

where = (—A^^B)*^, has a highly damped homogeneous 
term for practical values of its parameters. Note that we use 
upper scripts of time independent matrices to denote exponen¬ 
tiation. Eig. 3 presents | | as function of ft — k. The curves 

were computed for M = 512, although they are practically in¬ 
sensitive to M for M > > 1, and different values of a and S (with 
ry = r 2 = 1). As can be appreciated, the error introduced by a 
wrong estimation of the final condition, e.g. Y^ = Y^^^, can be 
reduced to negligible levels provided few frames at the end of 
the series are dropped. 

To study the overall conditioning of the restoration algo¬ 
rithm, with respect to uncertainties in the measured columns. 
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M=512, K=10 and a e {0, 4, 8, 12, 16, 20}% 



Fig. 4. Conditioning of the desmearing algorithm with respect 
to errors in the measured images, versus the parameters prod¬ 
uct MS. The curves correspond to a = 0,4,8,12,16 and 20% 
from bottom to top respectively. See Eq. 10 for extra details. 


let us write it using block matrices as follows (see Eq. 9), 


Y = UY + HY^, 


( 10 ) 


where. 


A B 0 ... 0 
0 A B ... 0 


B 0 0 ... A 


is a MK X MK, block circulant matrix [14]. Note Eq. 11 is linear 
in Y and thus it can be applied after averaging any number of 
periods, reducing this way the computational load. 

On one hand, numeric inversion of U will not be in general 
a problem because it has to be done only once for each setup, 
i.e. values of Si, S 2 and a. There are, however, Fourier-based 
methods used to invert block circulant matrices that can consid¬ 
erably reduce computation time and storage while increasing 
numeric stability [14,15]. 

On the other hand, the product in the right side of Eq. 11 
may become computationally expensive because it has to be 
performed once per image column. Two possible approaches 
to cope with this issue are: find only an approximate solution 
iteratively by first guessing Y® similar to Section 3A; or use the 
fact that is also block circulant and band dominant [15] to 
implement an efficient multiplication algorithm. 


where H = 

[(H^ 

r, (H«-l 

". 



A-1 

H^A 1 

H^A 1 ... 

1 

> 

1 

U = 

0 

A-i 

H^A 1 ... 



0 

0 

0 

A-i 


and 0 is the M x M null matrix. 

According to Eq. 10, when neglecting the contributions of 
HY^ (see Fig. 3), fhe propagation of relative errors in the mea¬ 
sured Y is upper bounded by the condition number of U, i.e. 
the ratio of fhe largesf fo the smallest singular values of U [13]. 
The latter is plotted in Fig. 4 versus MS (with ri = r 2 = 1) for 
M = 512, fC = 10 and different values of k. The curves are 
practically insensitive to M and K for M > > 1 and X > > 1. As 
expected, the restoration becomes monotonically worse condi¬ 
tioned for increasing S and a. 


B. Periodic iiiumination 

Of particular interest for this work is the solution of Eq. 5 to a 
time-periodic series of smeared columns, i.e. the illumination 
pattern repeats cyclically after a fixed amount of frames. As¬ 
suming periodicity in the scene allows the usage of image accu¬ 
mulation for noise reduction purposes. This is specially impor¬ 
tant for high-frame-rate applications that are photon starved. 

In contrast to Section 3A, in the periodic illumination sce¬ 
nario, image restoration can be performed for a single period of 

the input, Y = |^(Y'^)^, (Y^)^,..., , where X >26 

N is the period. The corresponding desmearing expression can 
be written in block matrix as, 

Y = tJ ^Y, (11) 


4. APPLICATION TO FAST IMAGING POLARIMETRY 

We applied the desmearing expression given in Section 3B to 
correct a set of images taken by the Fast Solar Polarimeter (ESP) 
[16]. We will not describe the instrument details here, only the 
relevant aspects related to the image smearing process. 

ESP uses a 264x264 pixels, split-frame-transfer pnCCD [17] 
that can record up to 800 frames per second (fps). The pnCCD 
operates in standard mode and its readout is synchronized with 
a four-states beam modulator [18] that will introduce an inten¬ 
sity variation according to the polarization state of the incom¬ 
ing light. The intensity switching profiles and synchronization 
between pnCCD and modulator meet the assumptions detailed 
in section 2 for Eq. 5. 

Any complete polarimetric measurement requires taking at 
least four consecutive exposures, one during each modulator 
state, that can have very different light levels for strongly polar¬ 
ized sources. The four recorded frames are linearly combined 
in a later step to retrieve the polarization signal. The sensor is 
constantly illuminated during measurements. If fhe incoming 
polarization state is constant, a periodic series of four images is 
obtained. 

The upper row of Fig. 5 presents an example measurement 
taken at 700 fps of a USAF target illuminated by a light source 
of constant linear polarization. Only a fraction of the total sen¬ 
sor area is shown. Each of the four exposures, denoted with 
labels State 1 to State 4, corresponds to an average of several 
measurement periods. Smearing is obvious in all images, note 
in particular its variable spatial distribution among different 
states. The latter can be easily explained using Eq. 5 and con¬ 
sidering that the mean light levels in the bright areas, i.e. above 
two sigmas from the image mean, of each state are very differ¬ 
ent. Namely 1950, 2828, 2825 and 297 arbitrary counts approxi¬ 
mately for sfate 1 to state 4 respectively. 

The lower row of Fig. 5 displays the restored images ob¬ 
tained after applying Eq. 11 with K = 4, M = 264, a = 3.9%, 
Si = 0.05% and S 2 = 0.03% to the images in the upper row. 
Reduction of the smearing is evident with only faint residuals. 
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Slate 1 State 2 State 3 State 4 



Fig. 5. Smeared (upper row) and corrected (lower row) im¬ 
ages, displayed using logarithmic scales to enhance the smear¬ 
ing. The gray scales of the images in each column are the same. 
The mean values of the bright areas in the upper-row images 
are 1950, 2828, 2825 and 297 arbitrary counts for sfates 1 to 4 
respectively. The dashed and continuous horizontal lines in 
states 3 and 4 images, delimit the cuts shown in Fig. 6 


(a) State 3 horizontal cut 




Pixels 

Fig. 6. Horizontal cut of the state 3 images (a) and vertical av¬ 
erage of a 4-pixel-wide horizontal stripe of state 4 images (b). 
The position of the cut and the center of stripe are indicated by 
the dashed and continuous, white lines in Fig. 5. In both plots 
the values for the smeared (dashed) and corrected (continu¬ 
ous) images, as well as their difference (dotted) are shown. All 
profiles are expressed in percentage of the mean value in the 
bright areas of the corresponding smeared image. 


The quality of the restoration and the differences in the arti¬ 
fact levels between state 3 and state 4 images, are exemplified 
by the profiles shown in Fig. 6. Firstly, from the corrected pro¬ 
files, note there is not clear residual smearing above the noise 
levels. Given the USAF target pattern, and in complete absence 
of smearing and stray light, the profiles of fhe selected cuts are 
expected to be flat except for noise induced variations. 

Secondly, from the smeared profiles in Fig. 6, note that the 
maximum artifact levels, relative to the mean in the bright ar¬ 
eas of each image, are larger in State 4 (~ 12%) than in State 
3 (~ 5%). To understand this, recall that each frame accu¬ 
mulates smearing signal during both the pre-exposure and the 
post-exposure frame transfers (see black boxes in Fig. 1). Fur¬ 
ther, note that the smearing signal acquired during the post¬ 
exposure transfer depends on the illumination level of fhe fol¬ 
lowing frame. In this way, smearing level in State 3 is low 
because the consecutive frame (state 4) has approximately 9.5 
times fainter illumination. On the contrary, the artifact level in 
State 4 is high because the consecutive frame (State 1, recall the 
input is periodic) has approximately 6.6 times larger intensity. 

5. CONCLUSIONS 

We developed a model for the smearing, introduced in frame 
transfer CCDs, that accounts for variations in the sensor illumi¬ 
nation provided the changes take place in synchronization with 
the detector readout and have a transition profile that is an odd 
function. The derived model depends only on three physical 
parameters, namely the period of the charge transfer clock, the 
image exposure time and the photon flux switching time. 

In addition, we showed that smearing introduces not only 
a spatial but also a temporal correlation in the images and de¬ 
rived a simple upper boundary for the noise degradation. 

We also studied the corresponding desmearing algorithm 
and its conditioning for the cases of non-periodic and periodic 
variable scenes. The latter was successfully applied to restore 
a set of polarimetric measurements taken by the Fast Solar Po- 
larimeter. 

The present work does not take in to account the effects of 
pixel saturation, pixel blooming, and charge transfer inefficien¬ 
cies. 

F. A. Iglesias would like to acknowledge the International 
Max Planck Research School for Solar System Science for sup¬ 
porting his participation in this project. 
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